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ABSTRACT 

To  ensure  network-centric  and  other  systems  provide  relevant  capability  to  the  user,  effort  needs 
to  be  expended  in  understanding  the  requirements,  designing  the  appropriate  solution, 
developing  the  capability  and  implementing  for  acceptance.  The  cognitive  systems  engineering 
and  the  software  systems  engineering  communities  struggle  with  the  difficulties  of  understanding 
a  domain  and  its  challenges  and  then  handing  research  results  to  designers  and  developers  so  that 
shared  understanding  of  the  problem  and  possible  solutions  exists.  They  also  struggle  with  the 
more  frustrating  challenge  of  a  developed  system  being  implemented  but  not  enthusiastically 
embraced  by  the  end-user.  Participatory  Design  (PD)  has  a  goal  of  engaging  researchers, 
designers,  developers,  practitioners  and  end-users  in  all  of  the  various  activities  leading  to  the 
successful  development  and  implementation  of  systems.  PD  is  an  umbrella  methodology  which 
includes  studies,  theories,  conferences  and  practices  (Muller  and  Kuhn,  1993;  Kensing  and 
Blomberg,  1998;  Madsen,  1999).  This  paper  will  discuss  two  methods  which  embrace 
participatory  concepts.  The  first  is  Elicitation  by  Critiquing  (EBC)  and  the  second  is  Value 
Elicitation.  Both  techniques  provide  a  way  to  engage  all  of  the  stakeholders  at  the  requirements 
discovery  stage  of  development,  which  is  the  first  critical  step  of  system  development. 

Introduction 

Cognitive  systems  engineers  (CSE)  and  software  systems  engineers  (SSE)  endeavor  to  develop 
useful,  usable  systems  to  support  cognitive  work  such  as  is  accomplished  in  a  network-centric 
environment.  Cognitive  task  analysis  (CTA)  methods  are  used  to  discover  expertise  that  domain 
practitioners  utilize  to  perform  tasks  so  that  better  support,  such  as  automation  or  training,  for 
these  cognitive  activities  can  be  developed.  Specifically,  CTA’s  identify  ineffective  strategies 
that  lead  to  poor  performance  (i.e.,  a  model  of  mistakes  that  “novices”  make),  as  well  as  adaptive 
strategies  that  have  been  developed  by  highly  skilled  practitioners  to  cope  with  task  demands 
(i.e.,  a  model  of  “expert”  skill).  The  complexity  of  understanding  the  environment  and  the  tasks, 
combined  with  the  fact  that  experts  performing  cognitive  tasks  have  difficulty  reliably 
articulating  about  the  task  when  asked,  contribute  to  making  discovering  expertise  hard.  There  is 
a  myriad  of  other  challenges  which  is  why  a  variety  of  cognitive  task  analysis  methodologies 
exist  (Schraagen  et  ah,  2000).  However,  many  of  these  methods  are  skeptically  viewed  by  a 
domain’s  practitioners  as  they  perceive  an  outsider  cannot  fully  understand  their  particular 
challenges. 
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In  addition,  despite  having  established  methods  of  gaining  understanding  about  a  domain,  the 
cognitive  systems  engineering  community  struggles  with  the  difficulties  of  handing  the  research 
results  to  software  designers  and  also  of  handing  designs  to  developers  so  that  shared 
understanding  of  the  problem  and  possible  solutions  exists.  Software  systems  engineers  (SSE ) 
are  also  busy  trying  to  support  user’s  needs  but  come  from  a  somewhat  different  angle  as  CSE 
focuses  on  the  cognitive  aspects  as  revealed  by  cognitive  and  domain  explorations  while  SSE 
focuses  on  issues  such  as  data  and  technology  requirements  and  concerns  itself  less  with  the 
cognitive  challenges  that  the  software  may  present.  While  software  systems  are  becoming 
increasingly  critical  and  vital  to  our  existence,  as  evidenced  by  the  trend  toward  net-centric 
environments,  these  two  communities  that  aim  to  support  the  user’s  needs  often  still  miss  the 
mark.  The  frustration  culminates  when  a  developed  system  is  implemented  but  is  not 
enthusiastically  embraced  by  the  end-user. 

Participatory  Design  (PD)  is  an  established,  diverse  research  and  practice  area  which  has  a  goal 
of  engaging  researchers,  designers,  developers,  practitioners  and  end-users  in  all  of  the  various 
activities  leading  to  the  successful  development  and  implementation  of  systems.  It  is  an  approach 
to  the  assessment,  design,  and  development  of  technological  and  organizational  systems  that 
places  a  premium  on  the  active  involvement  of  workplace  practitioners  (usually  potential  or 
current  users  of  the  system)  in  all  of  the  steps  in  the  research,  design  and  decision-making 
processes.  PD  is  an  umbrella  methodology  which  includes  studies,  theories,  conferences  and 
practices  (Muller  and  Kuhn,  1993;  Kensing  and  Blomberg,  1998;  Madsen,  1999).  When  utilized, 
PD  can  help  address  the  above  issues  that  the  CSE  and  SSE  communities  face  when  solving 
difficult  problems  as  the  entire  set  of  stakeholders’  viewpoints  are  brought  in  from  the  beginning. 
Elicitation  by  Critiquing  (EBC)  and  Value  Elicitation  are  two  participatory  methods  that  provide 
ways  of  engaging  the  stakeholders  in  the  requirements  discovery  stage  of  development  which  is 
the  first  critical  step  of  system  development. 

Elicitation  by  Critiquing 

Elicitation  by  Critiquing  (EBC)  is  a  methodology  that  engages  subject  matter  experts  themselves 
as  the  ones  to  reveal  strategies  of  another  domain  practitioner  to  the  CSE,  or  SSE,  investigator. 
First,  one  domain  practitioner  completes  a  task  and  then  that  completed  task  is  presented  to  yet 
another  domain  practitioner  for  critiquing  (Figure  1).  The  CSE  takes  note  of  the  critiques  to  gain 
and  document  understanding  of  the  domain  for  later  reflection  by  the  domain  expert. 

EBC  introduces  different  roles  for  the  CSE  practitioner  and  the  expert  than  other  cognitive  task 
analysis  methods  (Figure  2).  For  example,  with  direct  observation,  the  CSE  watches  the  expert  as 
a  domain  practitioner  performing  actual  or  simulated  tasks.  The  value  of  the  results  for  direct 
observation  is  mainly  a  function  of  how  realistic  the  scenarios  and  performances  are  and  how 
well  the  observed  events  stress  the  cognitive  system  in  order  to  reveal  leverage  points  for 
improvement.  With  interviewing,  the  expert  tells  information  to  the  CSE  investigator,  often 
stories  based  on  past  cases  and  experiences.  While  being  questioned  by  the  CSE,  the  expert  as 
storyteller  might  reveal  not  only  how  he  handled  a  particular  case,  but  also  may  have  further 
comments  on  how  that  case  changed  his  later  work  practices.  The  value  of  the  results  from 
interviews  is  mainly  a  function  of  the  probing  skill  of  the  investigator  and  how  well  the 
interviewee  understands  what  type  of  data  is  sought.  With  critiquing,  however,  the  domain  expert 
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evaluates  the  performance  of  another  domain  practitioner  thereby  participating  in  the  process  of 
revealing  knowledge  of  the  domain’s  practices  and  strategies.  This  has  the  added  benefit  of 
setting  up  a  situation  where  the  domain  expert  is  not  concerned  about  his  performance 
evaluation.  In  addition,  the  technique  relies  less  on  the  investigator’s  probing  skills  and  domain 
knowledge.  The  value  of  the  results  from  EBC  is  mainly  from  the  participation  of  the  domain 
practitioner.  EBC  also  has  the  benefit  of  combining  many  of  the  advantages  from  performance 
simulations  with  the  practical  advantages  of  interviewing  techniques. 

A  study  was  performed  to  determine  the  usefulness  of  EBC  (Miller  et  al,  2005).  The  goal  of  the 
elicitation  was  to  identify  military  intelligence  analysts’  strategies  and  challenges  encountered  in 
doing  their  work.  The  method  involved  two  stages.  First,  an  actual  trainee  performed  a  typical 
domain  task  in  order  to  capture  his  performance  process.  Then,  a  representation  (transcript, 
screen  shots  and  note-taking  artifact)  of  the  trainee’s  performance  on  the  task  was  presented  to 
six  experienced  domain  practitioners  for  critiquing.  The  experienced  practitioners'  comments  on 
the  trainee’s  performance  were  captured  and  iteratively  analyzed  for  patterns.  To  investigate 
how  much  the  critiquer’s  comments  were  influenced  by  the  performance  of  the  trainee,  a  second 
trainee  performed  the  same  task  and  was  critiqued  in  the  same  manner  by  two  of  the  original 
expert  critiquers. 

This  initial  investigation  into  critiquing  suggested  that  CTA  challenges  of  gaining  access  to  tasks 
and  experts,  efficiency,  and  repeatability  can  be  addressed  with  critiquing.  Task  accessibility 
issues  were  addressed  as  once  each  trainee’s  process  was  captured,  the  re-created  performances 
which  consisted  of  the  transcript,  the  screen  shots  of  the  querying  capability  and  slides  of  the 
note  artifact,  were  used  multiple  times.  They  were  used  six  times  for  the  first  trainee’s  process 
and  twice  for  the  second  trainee’s  process.  Hence,  as  the  task  was  packaged,  it  was  accessible  for 
multiple  sessions.  Physical  access  to  experts  was  overcome  because  the  critiquing  sessions  were 
held  outside  of  normal  work  areas,  which  have  security  access  issues.  Additional  access 
challenges  because  of  the  existence  of  only  a  few  experts  could  be  overcome  with  this  method  if 
several  trainees  or  novices  were  used  to  create  different  presentations  on  the  same  or  different 
problems  and  the  few  available  experts  critiqued  the  different  trainee  or  novice  processes,  as  was 
done  with  two  of  the  experts  critiquing  both  trainees.  Another  access  issue  that  arises  in  some 
domains  is  that  experts  are  reluctant  to  participate  due  to  repercussions,  such  as  erroneously 
performing  an  act  while  being  observed  that  could  have  dire  consequences.  For  Elicitation  by 
Critiquing,  as  the  expert  is  in  a  role  other  than  performing  work,  he  may  see  the  elicitation  as  less 
of  a  threat  and  therefore  be  more  willing  to  participate. 

While  the  above  listed  CTA  challenges  are  addressed  by  EBC,  challenges  still  exist  in  fully 
understanding  certain  types  of  problems  in  domains.  For  example,  repetitive  acts  or  processes, 
such  as  building  an  air  tasking  order,  can  be  investigated  with  EBC  but  uncovering  what 
instigates  an  ‘aha  moment’  for  creative  problem  solving,  such  as  a  commander  determining  an 
innovative  defense  strategy,  is  still  not  simple.  For  EBC,  such  a  moment  would  have  to  be 
created,  and,  hence,  what  made  that  moment  would  already  have  to  be  understood.  In  addition, 
if  the  domain  is  dynamic  so  that  new  methods  are  constantly  evolving,  such  as  in  basic  research, 
there  will  still  remain  some  mystery  to  the  domain.  In  these  cases,  another  participatory  method, 
value  elicitation,  can  be  used  to  help  identify  requirements  for  what  might  be  helpful  especially 
in  terms  of  training  and  software  tools. 
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Value  Elicitation 


Value  focused  thinking  (VFT)  is  a  proven  decision  analysis  methodology  that  can  be  applied  to  a 
variety  of  multi-criteria  situations  (Kirkwood,  1997).  The  methodology  concentrates  on  eliciting 
the  values  of  the  stakeholders  at  the  core  of  a  decision  before  any  particular  solution  to  the 
decision  is  considered.  As  an  objective  methodology,  VFT  is  well  suited  for  disclosing  and 
handling  multiple,  competing  requirements  of  the  stakeholders,  such  as  those  encountered  in 
interface  design  for  C2  and  other  network-centric  systems,  and  provides  an  unbiased  evaluation 
artifact.  The  primary  benefit  that  VFT  provides  is  its  ability  to  convert  the  goals  of  a  project  or 
values  of  an  organization  into  an  objective  realm  and  its  structure  lends  itself  to  handling  multi- 
objective  problems  even  if  the  objectives  are  of  a  subjective  nature.  Using  VFT,  high  level 
objectives  are  broken  down  into  smaller  values  during  facilitated  sessions  with  the  stakeholders. 
Once  articulated,  the  values  can  be  measured  and  put  to  a  common  scale,  allowing  their 
contribution  to  the  overall  objective  to  be  evaluated.  By  assigning  quantifiable  measurements  to 
the  components,  the  multi-objective  goal  can  be  evaluated.  Once  developed  the  hierarchy  can  be 
consistently  and  constantly  applied  to  different  system  developments  and  can  allow  a  fair 
comparison  between  potential  software  solutions. 

During  discussions  and  value  elicitations  with  the  stakeholders,  the  values  and  measures  are 
developed  and  placed  into  a  hierarchical  structure.  They  are  then  weighted  by  the  decision¬ 
makers.  The  weighing  process  allows  the  general  process  to  be  customized  to  the  particular 
instantiation.  The  interface,  or  other  object,  to  be  evaluated  is  then  scored  and  ranked  using  an 
additive  value  function,  producing  a  measure  of  derived  directly  from  the  decision-maker’s 
values.  The  additive  function,  v(x)  =  £A,ivi(xi)  for  all  i  measures,  associated  with  VFT 
methodology,  is  used  and  mutual  preferential  independence  is  assumed  (Kirkwood,  1997). 

As  VFT  concentrates  on  determining  the  values  at  the  core  of  the  decision,  the  choice  is  not 
between  varieties  of  alternatives — each  with  its  own  benefits  and  drawbacks — but  rather 
between  a  selection  of  the  alternatives  that  give  the  greatest  utility  with  regard  to  what  has  been 
determined  valuable.  VFT  emphasizes  that  values  should  be  the  focus  for  making  a  good 
decision.  However,  most  people  try  to  look  at  all  the  alternatives  and  compare  them  against  each 
other  without  defining  a  similar  basis.  This  presents  difficulty  if  one  alternative  is  extremely 
better  at  one  aspect  of  the  decision  while  the  other  alternative  is  extremely  better  at  another 
aspect.  This  type  of  decision  is  called  a  multi-objective  decision,  where  multiple  objectives  are 
desired  in  the  decision.  VFT  provides  a  structure  to  compare  these  objectives  against  each  other 
based  on  the  decision-maker’s  values.  VFT,  however,  takes  more  time  and  requires  the 
decision-maker  to  give  his  mind  to  the  exercise,  but  the  benefit  of  this  structure  makes  it  worth 
the  effort.  (Keeney,  1992)  In  addition,  as  it  is  participatory,  the  stakeholders  are  the  ones 
driving  the  development  of  the  hierarchy. 

In  previous  work,  McGee  (2003)  developed  a  VFT  hierarchy  (Figure  3)  during  discussions  with 
military  intelligence  analyst  participants.  This  hierarchy  was  then  used  to  evaluate  a  system 
which  was  still  under  development  for  that  military  intelligence  analyst  community.  As  expected, 
when  this  hierarchy  was  applied  to  the  newly  developed  system,  the  system  under  evaluation  did 
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not  fair  well,  scoring  only  0.13  out  of  a  possible  1.0  with  the  analysts  agreeing  with  this  scoring. 
One  relevant  outcome  of  this  exercise  was  that  the  evaluation  revealed  user  concepts  and  ideas 
that  the  developers  had  not  even  entertained. 

That  work,  presented  at  the  2004  Command  and  Control  Research  and  Technology  Symposium, 
on  VFT  (Miller,  2004)  has  been  extended  by  applying  the  hierarchy  to  an  additional  system  for 
evaluation.  The  VFT  hierarchy,  which  has  been  put  into  an  Excel  spreadsheet  for  computational 
ease,  was  applied  to  a  text  exploitation  capability  being  developed  for  the  same  intelligence 
analyst  community.  The  program  manager  for  the  system  development  had  become  sensitive  to 
human  factors  issues  once  the  foundations  of  the  technology  were  well  underway  and  was 
interested  in  knowing  how  the  tool’s  interface  would  fare.  She  had  worked  closely  with  the 
intended  user  community  during  the  majority  of  the  development  to  avoid  erroneous  assumptions 
and  wanted  to  verify  that  she  was  on  target.  When  the  capability  was  scored  by  the  intended  user 
community  using  McGee’s  VFT  model,  a  value  of  .588  was  derived.  While  this  showed  need 
for  further  usability  development,  the  text  exploitation  tool  was  still  in  early  enough  stages  of 
development  to  make  adjustments.  The  weak  areas  which  the  results  highlighted  were  not  totally 
surprising  as  the  program  manager  had  held  meetings  where  some  of  the  tool’s  interface 
shortfalls  had  been  discussed.  However,  this  score  raised  questions  on  the  limited  value  of  some 
aspects  of  the  existing  hierarchy.  Discussions  with  the  military  analysts  indicated  agreement  that 
there  was  more  refinement  to  be  done  on  the  hierarchy.  To  address  the  issues,  the  hierarchy  was 
further  developed  to  include  what  could  be  considered  a  ‘gold  standard’  (Figure  4).  The 
capability  was  then  rescored  and  received  a  higher  score,  this  time  a  .658,  which  helped  the 
program  manager  better  focus  in  on  requirements  for  improvement. 

The  above  usages  of  the  VFT  indicated  that  there  is  value  in  using  the  hierarchy  to  gain 
understanding  of  a  domain  and  its  requirements.  Nevertheless,  it  is  understood  that  any  model 
must  necessarily  be  based  on  some  type  of  particular  problem  set  which  causes  a  natural 
bounding  of  the  resulting  model  and  those  boundary  conditions  must  be  appreciated  when 
applying  the  model.  This  does  not  negate  the  value  of  this  multi-attribute  theory-based  model  for 
C2  or  other  domains  but  rather  the  method  can  be  added  to  the  tricks  of  the  CSE  and  SSE  trades 
for  identifying  requirements  for  development  of  tools  that  fully  support  the  human’s  role  to 
prevent  human-error-to-blame  mishaps. 

Discussion 

While  software  systems  are  becoming  increasingly  critical,  complicated  and  vital,  the  two 
communities  that  aim  to  support  the  user’s  needs  struggle  with  the  difficulties  of  understanding 
and  articulating  about  domains  and  their  challenges  so  that  excellent  systems  are  implemented. 
CSE  focuses  on  the  cognitive  aspects  as  revealed  by  cognitive  and  domain  explorations.  SSE 
focuses  on  issues  such  as  data  and  technology  requirements  and  concerns  itself  less  with  the 
cognitive  challenges  that  the  software  may  present.  A  well-accomplished  CTA,  a  critical  step  in 
system  development,  indeed  reveals  critical  sources  of  cognitive  complexity  that  must  be 
addressed  in  new  systems,  but  the  hand-off  to  the  software  designers  and  developers  is  often  in 
the  form  of  staid  documents  and  reports.  While  both  disciplines  are  necessary  to  create  systems, 
the  partnering  of  the  two  has  been  irregular  and  awkward  leaving  a  gap.  In  addition,  the  domain 
practitioners  are  often  not  brought  into  the  decision  making  process  during  system  development. 
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As  a  result,  both  communities  and  the  domain  practitioners  experience  frustration  when  an 
implemented  system  misses  the  mark  either  in  its  capabilities  or  its  ability  to  be  embraced  for 
other  reasons. 

Domain  practitioners  want  systems  that  address  their  needs  as  much  as  the  CSE  and  SSE 
communities  want  to  develop  such  systems.  Participatory  Design  is  an  established  category  of 
methods  which  aim  to  inform  and  engage  all  stakeholders  in  the  process  of  system  development. 
Involving  the  expected  users  and  various  stakeholders  during  requirement  definition,  as  is 
advocated  by  PD,  is  important  for  developing  useful,  usable  systems  which  gain  acceptance 
when  implemented.  Both  EBC  and  Value  Elicitation  leverage  the  knowledge  and  understanding 
of  potential  users  of  a  system  by  having  them  participate  in  understanding  the  requirements  for 
support  before  a  system  is  built.  These  methods  contribute  to  the  pool  of  potential  PD  methods 
and  should  help  the  CSE  and  SSE  communities  work  together  for  the  benefit  of  the  domain 
practitioners  to  overcome  the  gaps  between  task  analysis,  design  and  development  which  are 
arise  in  building  C2  and  other  systems. 
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Figure  4.  Gold  Standard  for  Software  Interfaces 
DISCLAIMER 

The  views  expressed  in  this  paper  are  those  of  the  author  and  do  not  reflect  the  official  policy  or 
position  of  the  United  States  Air  Force,  Department  of  Defense  or  the  U.  S.  Government. 
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Problem 


*  In-depth  understanding  of  domain  is  required  for 
developing  accurate,  training  and  support  for  net-centric 
environments  relevant 


•  Cognitive  systems  engineers  (CSE)  and  software 
systems  engineers  (SSE)  endeavor  to  develop  useful, 
usable  systems  to  support  cognitive  work  such  as  is 
accomplished  in  a  network-centric  environment 

•  Participatory  Design  (PD)  is  an  established,  diverse 
research  and  practice  area  which  has  a  goal  of  engaging 
researchers,  designers,  developers,  practitioners  and 
end-users  in  all  of  the  various  activities  leading  to  the 
successful  development  and  implementation  of  systems. 


Why  does  tacit  knowledge  need  to  be  captured? 

Tacit  knowledge  -  what  walks  out  the  door  ->  how  to  apply 

‘Just  do  it’  ->  Capt  John 

Domain  of  interest:  Intelligence  analysts,  specifically  looked  at  National  Air  Intelligence  Center 
Complicating  factors  are  not  atypical  to  commercial,  other  sectors: 

Change  in  work  focus,  people  leaving,  retiring,  transferring 
Simon  and  Chase  (1973)  10  years  to  reach  high  level  of  performance 
Ground  in  context  ->  gives  meaning 
Accessibility:  to  work  areas,  to  people  willing  to  share 

tasks  that  have  meaning 

Laborious  and  time  consuming  the  bottleneck  of  knowledge  acquisition 

Perhaps  10  hours  to  transcribe  and  analyze  each  hour 
Repeatability  ->  to  compare  results 

Two  parts  to  question: 

General:  Can  the  critiquing  method  be  used 

Specific:  Unveil  expertise  in  intelligence  analysis 
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Potter,  Woods  -  Bootstrapping 

Goal  of  bootstrapping  was  to  look  at  contributions  of  different  methods  to  see 
how  they  supported  end-to-end  development  of  support.  The  selection  and 
timing  of  particular  techniques  to  be  deployed  will  depend  on  the  detailed 
constraints  and  pragmatics  of  the  particular  domain  being  addressed. 

Looking  at  the  picture,  we  see  several  balancing  areas: 

Exploring  the  current  world  while  exploring  the  envisioned  world 

Understanding  the  Practitioner  and  how  he  works  in  his  world  and 
understanding  the  domain  and  how  the  world  works. 

As  we  expand  our  knowledge  over  time,  we  should  be  focusing  in  on  particular 
ways  to  help 


14 


Big  picture  needs  to  be  considered.  If  we  are  to  change  the  way  people  work  in 
the  long  term,  we  cannot  just  look  at  each  task  separately  but  consider  the 
particular  while  looking  at  the  general  contribution.  Few  things  pertain  to  one 
task,  one  domain.  CTA  is  a  means  to  a  larger  end  of  specifying  ways  to  improve 
human  and  team  performance  in  the  domain.  From  this  perspective  CTA  is  bet 
thought  of  as  a  process  for  uncovering  the  cognitive  activities  entailed  by  the 
field  of  practice  an  identifying  opportunities  for  more  effective  support. 

Authentic:  Need  realistic  data,  criticism  of  Tab’  studies.  Study  constraints 
made  results  not  applicable. 

Abstract:  Research  base  cannot  always  be  emptied  but  must  be  filled  with  ideas 
on  patterns 

Generative:  CTA  is  not  done  just  to  understand  but  the  goal  is  to  support  and 
build  for  support  for  people  who  are  trying  to  do  a  good  job.  Systems,  training, 
and  add  to  research  base 

Participative:  Technology  cannot  be  thrown  over  the  wall  and  the  human 
expected  to  adapt  and  learn.  Prototypes  are  dangerous  unless  the  going  in 
position  is  ‘this  might  not  be  right’  Often  the  prototype  begins  to  be  THE 
solution  and  the  questions  and  actions  are  changed  to  fit.  No  longer  is  the 
viewpoint  ‘what  can  be  made  to  support  the  user’  but  rather  ‘How  can  we  get 
this  prototype  to  fit  a  need’ 
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Discussion  of  EBC 


•  Remember:  Any  method  shapes  the  conditions  of 
observation. 

•  Relationship  to  C2 

-  Can  be  used  in  conjunction  with  Modeling  & 
Simulation 

-  Provides  strong  cue  to  focus 
—  Participatory  role 


1.  Considerations: 

Scenario  chose/presentation  method  shape  the  conditions  of  observation. 

What  needs  to  be  observed,  make  sure  object/event  of  interest  is  observable 
Be  aware  of  introducing  biases,  prior  knowledge,  including  typical  errors 

2. Appropria  te  domains. 

•  Domain  where  experts  may  be  reluctant  to  be  directly  observed  cue  to 
risks 

•  Need  for  strong  cue  to  remember  past  experiences 

•  Want  to  emphasize  being  participatory 
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10  Steps 


1.  Define  Problem 

2.  Build  Hierarchy 

3.  Identify  Evaluation  Measures 

4.  Establish  Evaluation  Functions 

5.  Weighting 

6.  Choose  Alternatives  to  Evaluate 

7.  Score  Alternatives 

8.  Deterministic  Analysis 

9.  Sensitivity  Analysis 

10.  Analyze  Conclusions 
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VFT  with  intel  Analysts 
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Discussion  on  Value  Elicitation 


Uncovers  Hidden  Objectives 
Improving  Communication 
Guides  Information  Collection 
Facilitates  Involvement  in  Multiple 
Stakeholder  Decisions 
Guides  Strategic  Thinking 
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Conclusion 


•  Domain  practitioners,  CSEs  and  SSEs  want  systems  that 
address  their  real,  understood  needs 

•  Participatory  Design  -  established  category  of  methods  which 
aim  to  inform  and  engage  all  stakeholders  in  the  process  of 
system  development. 

•  Involving  the  expected  users  and  various  stakeholders  during 
requirement  definition  is  important  for  developing  useful,  usable 
systems  ->  gain  acceptance  when  implemented 

•  Both  EBC  and  Value  Elicitation  leverage  the  knowledge  and 
understanding  of  potential  users  of  a  system  by  having  them 
participate  in  understanding  the  requirements  for  support 
before  a  system  is  built. 
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